Could you fix the CI failures?

[info] *** 24 TESTS FAILED ***
[error] Failed: Total 4431, Failed 24, Errors 0, Passed 4407, Ignored 28, Canceled 6
[error] Failed tests:
[error] 	org.apache.spark.deploy.history.RocksDBBackendHistoryServerSuite
[error] 	org.apache.spark.deploy.history.LevelDBBackendHistoryServerSuite
[error] (core / Test / test) sbt.TestsFailedException: Tests unsuccessful

@dongjoon-hyun Thank you for your interest.

What happens when there exists a conflict among the log directories? For example, a user want to abuse this as a kind of multi-tier log managements like the following and copy from shorterm to longterm? Of course, the sync operation is non-atomic.

hdfs://spark-events/shorterm
hdfs://spark-events/longterm

Each event log file is tracked by its full path as the key in LogInfo. So if the same application's event log exists in both directories, they are treated as separate entries.
I didn't anticipated such kind of usage but during a non-atomic copy, the incomplete log file in the destination directory may fail to parse or show incomplete information temporarily. However, on the next scan cycle, shouldReloadLog invoked through checkForLogs detects the file size change and re-parses it, so the entry self-corrects once the copy completes.

What is the semantic on the ordering in the config value? Especially, when we have SPARK-52914 ?

The ordering of directories in the config value has no semantic. All directories are scanned equally in each polling cycle (checkForLogs iterates over all logDirs). The order does not affect priority.

On-demand loading operates per log file within checkForLogsInDir, which is called independently for each directory. There is no cross-directory interaction, so I believe multiple directories support and on-demand loading are orthogonal and work together without issues.

Thank you. This is a nice feature. I'll try to test more seriously.

@dongjoon-hyun Thank you for your feedback.

Do you want to have per-directory configurations in the future?

I considered it might be helpful to have per-directory configurations (e.g. spark.history.fs.cleaner.*) but this such configurations are not supported at least in this PR, and I'd like to start with simple global settings and improve based on user feedback.

For now, spark.history.fs.update.interval is supposed to be applied for one scan for all directories?

Yes.

spark.history.fs.cleaner.interval is also supposed to be applied for one scan for all directories?

Yes.

When spark.history.fs.cleaner.maxNum is applied,
This PR will consider the total number of files for all directories, right?
Which directory will be selected as a victim for the tie?

Yes, the property is applied to the total number of log entries across all directories. As the updated document says, when the limit is exceeded, the oldest completed attempts are deleted first regardless of which directory they belong to.

Since this introduces lots of ambiguity a little, could you revise the PR title and provide a corresponding documentation update, docs, together in this PR?

Updated (You said revise the PR title but I thought it's type for PR description so I've updated only the description).

Thanks @yaooqinn for your feedback. I've updated.

@sarutak I found one potential correctness issue:

In FsHistoryProvider, AttemptInfoWrapper currently stores attempt.logPath as basename (reader.rootPath.getName()), and later operations resolve it via resolveLogPath(attempt.logPath) by scanning configured log dirs and returning the first match.

This may be ambiguous when two configured directories contain the same event log basename (e.g. after migration/copy or multi-cluster aggregation，although the probability is very low.). In that case, download/rebuild/cleanup may operate on a different physical file than the one originally indexed.

For example, the following test case：

test("same log file name across directories resolves incorrectly") {
    val dir2 = Utils.createTempDir(namePrefix = "logDir2")
    try {
      val conf = createTestConf()
        .set(HISTORY_LOG_DIR, s"${testDir.getAbsolutePath},${dir2.getAbsolutePath}")
      val provider = new FsHistoryProvider(conf)

      val collidingLogName = "shared-event-log"
      val log1 = new File(testDir, collidingLogName)
      writeFile(log1, None,
        SparkListenerApplicationStart("app1", Some("app1-id"), 1L, "test", None),
        SparkListenerApplicationEnd(5L))
      val log2 = new File(dir2, collidingLogName)
      writeFile(log2, None,
        SparkListenerApplicationStart("app2", Some("app2-id"), 2L, "test", None),
        SparkListenerApplicationEnd(6L))

      updateAndCheck(provider) { list =>
        list.size should be (2)
        list.map(_.id).toSet should be (Set("app1-id", "app2-id"))
      }

      val attempt = provider.getAttempt("app2-id", None)
      attempt.logSourceFullPath should be (dir2.getAbsolutePath)

      val resolveLogPath = PrivateMethod[(FileSystem, Path)](Symbol("resolveLogPath"))
      val (_, resolvedPath) = provider invokePrivate resolveLogPath(attempt.logPath)
      resolvedPath.toUri.getPath should be (log2.getAbsolutePath)

      provider.stop()
    } finally {
      Utils.deleteRecursively(dir2)
    }
  }

In a scenario where multiple log directories are configured in SHS and there are event log files with the same name in different directories, I expect the log path of app2 to be resolved to the second directory where it actually resides. However, in practice, resolveLogPath matches the file with the same name in the first directory first, resulting in an incorrect path resolution.

So can we assume that filenames are globally unique across configured directories?

@LuciferYang Thank you for pointing it out. While event log filenames are typically unique in normal operation, duplicates can occur during migration or multi-cluster log aggregation.
I've fixed resolveLogPath to use the source directory hint (logSourceFullPath) that is already recorded during indexing:

1. When logSourceFullPath is available, the method first checks that specific directory
2. If the file is not found there (e.g., file was moved), it falls back to scanning all directories

I also added a test based on what you show.

@dongjoon-hyun Thank you for your suggestion. I'll open a PR for some more test coverage.

Thank you! Feel free to merge this PR first so that we can get more feedbacks from the community, @sarutak .

Merged to master. Thank you @dongjoon-hyun @yaooqinn @LuciferYang the reviews and feedbacks!